Primal-dual coding to probe light transport

We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene's transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.Alfred P. Sloan Foundation (Research Fellowship)United States. Defense Advanced Research Projects Agency (DARPA Young Faculty Award)Massachusetts Institute of Technology. Media Laboratory (Consortium Members

Learning Lens Blur Fields

Optical blur is an inherent property of any lens system and is challenging to model in modern cameras because of their complex optical elements. To tackle this challenge, we introduce a high-dimensional neural representation of blur$-$$\textit{the lens blur field}$$-$and a practical method for acquiring it. The lens blur field is a multilayer perceptron (MLP) designed to (1) accurately capture variations of the lens 2D point spread function over image plane location, focus setting and, optionally, depth and (2) represent these variations parametrically as a single, sensor-specific function. The representation models the combined effects of defocus, diffraction, aberration, and accounts for sensor features such as pixel color filters and pixel-specific micro-lenses. To learn the real-world blur field of a given device, we formulate a generalized non-blind deconvolution problem that directly optimizes the MLP weights using a small set of focal stacks as the only input. We also provide a first-of-its-kind dataset of 5D blur fields$-$for smartphone cameras, camera bodies equipped with a variety of lenses, etc. Lastly, we show that acquired 5D blur fields are expressive and accurate enough to reveal, for the first time, differences in optical behavior of smartphone devices of the same make and model

Approximate N-View Stereo

. This paper introduces a new multi-view reconstruction problem called approximate N-view stereo. The goal of this problem is to recover a oneparameter family of volumes that are increasingly tighter supersets of an unknown, arbitrarily-shaped 3D scene. By studying 3D shapes that reproduce the input photographs up to a special image transformation called a shuffle transformation,we prove that (1) these shapes can be organized hierarchically into nested supersets of the scene, and (2) they can be computed using a simple algorithm called Approximate Space Carving that is provably-correct for arbitrary discrete scenes (i.e., for unknown, arbitrarily-shaped Lambertian scenes that are defined by a finite set of voxels and are viewed from N arbitrarily-distributed viewpoints inside or around them). The approach is specifically designed to attack practical reconstruction problems, including (1) recovering shape from images with inaccurate calibration information, and (2) building ..

Affine surface reconstruction by purposive viewpoint control

We present an approach for building an affine representation of an unknown curved object viewed under orthographic projection from images of its occluding contour. It is based on the observation that the projection of a point on a curved, featureless surface can be computed along a special viewing direction that does not belong to the point’s tangent plane. We show that by circumnavigating the object on the tangent plane of selected surface points, we can (1) compute two orthogonal projections of every point projecting to the occluding contour during this motion, and (2) compute the affine coordinates of these points. Our approach demonstrates that affine shape of curved objects can be computed directly, i.e., without Euclidean calibration or image velocity and acceleration measurements.